Optimization of the Architecture of Feed-forward Neural Networks with Hidden Layers by Unit Elimination

A method for red ucing t he number of uni ts in t he hidden layers of a feed-for war d neur al network is pr esented . Starting wit h a net that is oversize, t he redundan t unit s in t he hidden layer are elimina ted by introducing an addit ional cost fun ct ion on a set of auxiliary linear response uni ts . T he extra cost funct ion enables t he auxiliary units to fuse together the redundant uni t s on the original network, and t he aux iliary units serve only as an interm edi at e const ruct t hat vanishes when t he met hod converges. Nume rical tests on t he P arity and Symmetry problems illustrate t he usefu lness of t his method in pr actice.